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INTRODUCTION 

Software provides an increasingly larger part of the func- 
tionality, and consequently the cost, of computer systems. 
For some large applications, the cost of the software com- 
ponent exceeds 75 percent of the total system cost. The 
ability to deliver reliable software on time at minimum cost 
has become essential to success in the computer industry. 
The delayed first launch of the National Aeronautics and 
Space Administration (NASA) space shuttle clearly demon- 
strated the consequences of software failure. 

Software development organizations, therefore have 
strong incentives to Improve the software development 
process, principally by adopting new technology. A wealth 
of potentially beneficial software engineering tools, practices, 
and techniques has emerged in the past several years. Ma- 
ny. nowever, have been empirically evaluated (Reference 
1). Furthermore, experience shows that ail software en- 
gineering technologies are not appropriate for all software 
development problems and environments. 

The difficulty of accurately measuring the software de- 
velopment process, in general, and technology use, in par- 
ticular. accounts for much of the lack of objective informa- 
tion in this area. This paper describes an ongoing technolo- 
gy evaluation program (Reference 2) conducted by the Soft- 
ware Engineering Laboratory (SEL) that is intended to resolve 
these issues, at least in part. In the context of this paper, the 
term "technology" refers to tools, practices, and technique 
applied by software developers. 


Software engineering laboratory 

The SEL is a research project (Reference 3) sponsored 
by NASA and supported by Computer Sciences Corpora- 
tion and the University of Maryland. Figure 1 shows the or- 
ganization of the SEL. which was established in 1 977. The 
SEL studies software developed to support spacecraft flight 
dynamics applications at Goddard Space Flight Center. 

The overall objective of the SEL is to understand the soft- 
ware development process in the flight dynamics environment 
and the identify the ways in which it can be altered to im- 
prove the quality and reduce the cost of the product. The 
SEL has monitored the development of more than 45 fligth 
dynamics projects. In addition, the SEL conducts control- 
led experiments and performs multiproject variation studies. 


Right dynamics software 

The general class of spacecraft flight dynamics software 
studied by the SEL includes applications to support at: 'ude 
determination/ control, orbit adjustment, maneuver p'an- 
ning, and mission analysis. The attitude ground supper, sys- 
tems form a large and homogeneous group of software that 
has been studied extensively. Each system includes a te ’em- 
etry processor, data adjuster, and attitude computatio.n sub- 
systems as well as other necessary suporting functions. 

Flight dynamics applications are developed in FORTRAN 
on IBM mainframe comouter5;. Rystpm ql7P«5 rgnno 30 
to 1 50 thousand source lines of code. The fixed spacecraft 
launch date imposes a severe development time constraint. 
Acceptance testing must be completed 2 months prior to 
launch so that launch preparations can proceed on sched- 
ule. Figure 2 describes some major characteristics of ^i.:jht 
dynamics software. 


THE PROGRAM 

The SEL program of technology evaluation includes three 
steps: measurement, evaluation, and transference. Meas- 
urement establishes the baseline against which the ef-ects 
of technologies can be compared. Next, technologica' in- 
novations are attemped and their effects evaluated. After 
careful study, sucessful technologies are transferred to de- 
velopers via guidelines, standards, and training. 

Measurement 

Measurement is the basic prerequisite for technoiogy 
evaluation and management. Software engineeririg e:<perts 
such as Boehm (Reference 4) and DeMarco (Reference 5) 
are paying increased attention to the role of measurerr<=nt 
in software development. The SEL developed a comprer-en- 
sive data collection methodology (Reference 6) as the ba- 
sis for its measurement activity. Measures collected include 
staffing, computer utilization, error reports, and product 
size/complexity measures, as well as the level of technolo- 
gy applied to each project. The SEL employs both question- 
naires and automated methods of data collection. The col- 
lected data are assembled in a computerized data base ac- 
cessible to all SEL participants. 

Because the software development process is compiex 
and involves many different human and physical elemer ts. 
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many measures are needed to characterize it adequately. 
Examination of a model of software development, such as 
that shown in Figure 3, helps to define overlapping set of 
measures. This model includes the following components: 

— Problem — Statement of the information needs for which 
a software solution is desired 

— Personnel — Software development team, managers, 
and supporting personnel 

— Process — Practices, tools, and techniques employed 
by the personnel* to develop the product; it proceeds in 
a series of steps (the software life cycle) 

— Environment — Physical and informatbnal resources and 
constraints within which the personnel and process 
operate 

— Product — Software and documentation that solve the 
problem 

Measures are needed to characterize the principal attrib- 
utes of these components before the relationships among 
the components can be determined. A complete set of mea- 
sures constitutes a profile. Software Development profiles 
form the baselines against which technologies are evaluated. 

Evaluation 

Even after assembling a substantial software engineer- 
ing data base, other obstacles to accurate technology evalu- 
ation remain: technologies tend to be applied together, sam- 
ple sizes are small, and many nontechnoiogy factors also 
affect the outcome of a software development proiect. These 
complications prohibit a simplistic statistical analysis and in- 
terpretation of the software development measures collected 
(Reference 4), Nevertheless, some trends in the data are 
clear. 

The SEL has extensively studied four factors: program- 
mer performance, modern programming practices, quality 
assurance, and computer utilization. Figure 4 summarizes 
the results of an analysis of covariance performed with SEL 
data (Reference 7). Programmer performance proved to be 
the most important factor with respect to both productivity 
and reliability. 

Figure 5 shows range of programmer productivity values 
encountered In the SEL data. The figure indicates that vari- 
antbn is lessened (or performance is homogenized) in large 
projects. Figures 6, 7 and 8 plot data from 1 4 large attitude 
projects to illustrate the effects of the other factors 

Modem programming practices 

One group of individual technologies, referred to as 
modern programming practices, tend do be applied 
together. These techologies provide a flexible methodolo- 
gy for the (detailed) design, implementation, and verifica- 
tion of software. 

As practiced in the flight dynamics environment, the pnn- 
cipal components are as follows: 

— Informal program design language 

— Top-down development 

— Structured programming 

— Code reading 

— Strutured FORTRAN preprocessor 

The individual measures of technology use (listed above) 
were combined to form a single index of overall structured 
programming use. Figure 6 shows the relationship of this 
index to error rate. The use of modern programming prac- 
tices appears to be associated with a reduced error rate. No 
significant correlation with productivity was found. This im- 
plies that the reliability benefits of modern programming 
practices are obtained at no additional cost in terms of de- 
velopment effort. . .. 


Quality assurance 

Quality assurance includes all review and management 
procedures undertaken to ensure the delivery of an effec- 
tive and realible product. The specific technologies studied 
by the SEL are as follows: 

— Requirements Reviews 

— Design Reviews 

— Design Walkthroughs 

— Code Walkthroughs 

— Test Formalism 

— Test Followthrough 

~ Methodology Reinforcement 

— Document Quality Assurance 

— Development Standards 

— Code Configuration Control 

— Code Library (PANVALET) 

— Configuration Analysis Tool 

For the analysis decribed here, the individual measures 
of technology use (listed previously) were combined to form 
a single index of overall quality assurance activity. Figure 
7 shows the relationship of this index to error rate. Quality 
assurance activity appears to be associated with a reduced 
error rate. No significant correlation with productivity was 
found. This implies that the reliability benefits of quality as- 
surance are obtained at no additional cost in terms of de- 
velopment effort. 

Computer utflizetion 

Another major factor in software development is the com- 
puting environment. Because changes to this environment 
•jcually car.r.ct be made to cur uumi io the needs of any sin- 
gle project, the computing environment is not considered 
to be a software engineering technology as defined in this 
paper. Nevertheless, it can have a strong effect on the de- 
velopment process (Reference 8). The flight dynamics com- 
puting environment provides the programmer with facilities 
for editing, compiling, linking, and testing source code. 
Figure 8 shows that extensive computer use is associat- 
ed with low productivity. Heavy computer users may not 
spend enough time desk checking and planning their work 
before jumping into code and test. However, a recent study 
(Reference 9) indicated that computer support for design 
and planning activities (not now provided) can increase the 
overall productivity and reliability of the software develop- 
ment process. 

Transference 

When the effectiveness of a technology has been demon- 
strated, the next step is to transfer it to software developers. 
The principal mechanisms used by the SEL to accomplish 
technology transfer include disseminating guidelines, de- 
veloping tools, and conducting specialized training. The 
guidelines produced by the SEL cover management proce- 
dures (Reference 10), programming practices (Reference 
1 1 ), and quality assurance (Reference 1 2). Two important 
SEL-developed tools are the Source Code Analyzer Program 
(Reference 1 3), which has been distributed across the United 
States, and the Configuration Analysis Tool (Reference 14), 
which is tailored to specific flight dynamics needs. Current- 
ly, SEL researchers are designing a training program for the 
Ada* language (Reference 15). 


* Ada is a registered trademark of the U.S. Goverment, Ada Joint Pro- 
gram Office. 
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CONCLUSIONS 

SEL experience demonstrated that the software deveiop- 
ment process can be improved by a throughful program of 
technology evaluation. Other similar organizatbns can also 
apply the lessons learned by the SEL. First, the use of modem 
programming practices increases software reliability without 
noticeably increasing development cost. Second, a regular 
program of quality assurance also improves software relia- 
bility at little or no net cost. Whereas many modern program- 
ming concepts are firmly established in software engineer- 
ing practice, formal quality assurance procedures are only 
now coming into widespread use. Third, intensive computer 
use appears to be associated with tow productivity. Program- 
mers who spend a lot of time at the terminal tend to be less 
productive. 

In summary, these results suggest that a formal and con- 
. scientious method of software development yields a more 
reliable product. On the other hand, it is very difficult to 
reduce the cost of developing a software product, although 
a more reliable product should require less subsequent main- 
tenance. Despite technological advances, the major factor 
in both productivity and reliability continues to be person- 
nel capability and performance (Reference 16). 
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FIGURE 1 — Software Engineering Laboratory 
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FIGURE 2 — Flight Dynamics Software 
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FIGURE 3 — Software Development Model 
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FIGURE 4 — Technology Evaluations Summary 
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